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ABSTRACT 


Armed  forces  around  the  world  are  considering  radical  transformations  to  their  structures 
and  strategies  because  of  the  information  revolution  and  the  changing  global  environment. 
Senior  leadership  continually  face  decisions  on  how  best  to  structure,  modernize, 
organize,  and  employ  forces  in  an  increasingly  uncertain  future.  For  many  of  these 
problems  analytical  methods  are  not  applicable,  and  large-scale  experimentation  is  not 
feasible.  Simulation  provides  a  valuable  tool  for  addressing  these  types  of  problems.  One 
key  characteristic  of  these  decisions  is  the  large  number  of  factors,  and  interactions 
between  factors  that  impact  decision  makers.  Traditional  simulation  approaches  are  not 
designed  to  deal  with  this  many  factors,  therefore  the  results  are  often  incorrect  or 
misleading.  In  this  paper  we  introduce  and  implement  efficient  design  of  experiments 
techniques  to  analyze  C2  organizational  models  and  pursue  optimal  settings  for  different 
performance  measures.  This  allows  analysts  to  rapidly  identify  the  important  factors 
within  the  simulation,  employ  an  experimental  design  to  fully  explore  the  simulation 
space  efficiently,  and  design  the  systems  with  desired  optimal  performances  with  the 
simulation  model.  This  effort  dramatically  increases  the  breadth  and  depth  of  insights 
possible  when  the  simulation  output  data  are  analyzed,  while  reducing  the  time  required 
for  performing  a  study. 


1.  INTRODUCTION 


In  Joint  Vision  2010,  Army  General  John  M.  Shalikashvili,  Chainnan  of  the  Joint  Chiefs 
of  Staff,  said  “The  nature  of  modern  warfare  demands  that  we  fight  as  a  joint  team.  This 
was  important  yesterday,  it  is  essential  today,  and  it  will  be  even  more  imperative 
tomorrow.”  In  light  of  the  critical  transfonnation  drive  of  the  U.S.  military,  innovative 
organizational  structures  are  needed  to  deliver  better  team  performance. 

How  can  appropriate  organizational  structures  be  identified?  One  way  is  via 
leveraging  subject-matter  expertise  to  identify  a  few  specific  alternatives  expected  to 
yield  better  performance  in  complex,  dynamic  environments;  perfonn  the  training 
necessary  for  teams  to  operate  effectively  under  these  organizational  structures;  and  then 
conduct  some  live  experiments  or  exercises  to  assess  whether  the  new  structures  are, 
indeed,  effective.  However,  the  costs  and  time  are  prohibitive  for  any  broad-scale 
investigation.  It  is  possible  (even  likely)  that  the  subject  matter  experts  may  not  be  able 
to  identify  the  “best”  organizational  structures  a  priori — precisely  because  the 
environment  is  so  complex.  This  means  that  even  if  the  physical  exercises  indicate  that 
improvements  in  overall  perfonnance  are  possible,  these  improvements  might  be  far 
below  the  “optimal”  levels  of  performance  that  could  be  achieved. 

An  alternative  approach  involves  experimenting  on  computational  models  of 
organizational  perfonnance — such  as  organizations  with  different  levels  of  “edginess.” 
The  basic  idea  is  to  save  time  and  money  by  conducting  experiments  in  the  virtual 
domain.  The  identification  of  the  most  important  drivers  of  organizational  perfonnance 
can  provide  decision  makers  with  much  richer  insights  into  how  to  deliver  better  team 
performance. 

Researchers  using  simulation  to  explore  organizational  perfonnance  have  coined 
the  term  “computational  organizational  theory”  for  this  type  of  exploration.  Yet  even  in 
the  simulation  environment,  the  studies  have  typically  been  limited  to  a  handful  of  factors 
or  scenarios.  But  the  simulation  environment  offers  challenges  and  opportunities — the 
best  designs  are  typically  not  familiar  to  those  who  conduct  field  or  laboratory 
experiments  involving  human  subjects  (Kleijnen  et  al.,  2005).  Keys  to  effective 
simulation  studies  are  efficient  experimental  designs  and  interfaces/infrastructure  for 
automating  experiments.  In  this  way,  the  power  of  efficient  experimental  designs  is 
available  to  an  analyst  without  requiring  the  analyst  to  develop  expertise  in  the  technique. 

In  this  paper,  we  describe  our  recent  efforts  to  enhance  the  infrastructure  for 
performing  large-scale  simulation  studies  on  models  of  organizational  perfonnance.  Our 
primary  goal  is  to  assist  Edge  researchers  perfonn  large-scale  simulation  studies  using 
efficient  experimental  designs.  The  approach  requires  a  change  in  mindset  for  many 
researchers  and  analysts  who  otherwise  might  not  think  of  exploring  more  than  a  handful 
of  factors  or  scenarios,  (see,  e.g.,  Carley  and  Glasser,  1999;  Nissen  and  Levitt,  2004). 
Yet  the  benefits  can  be  dramatic. 


The  importance  of  computational  experimentation  for  the  analysis  of  these  complex 
systems  is  discussed  in  Gateau  et  al.  (2007).  In  the  paper,  the  relative  multidimensional 
performances  of  six  theoretically  distinct  organizational  forms  are  compared,  but  no  strict 
analysis  of  each  factor’s  (parameter’s)  effect  on  the  performance  has  been  stressed 
(although  some  initial  observations  have  been  made).  As  the  authors  state, 

“...  if  we  can  identify  the  model  parameters  that  enable  the  Simple  Structure 
to  keep  risk  below  that  of  the  Machine  Bureaucracy  (e.g.,  formalization),  that 
enable  the  Professional  Bureaucracy  to  operate  so  quickly  in  predictable 
environments  such  as  the  Industrial  Era  (e.g.,  application  experience),  and 
that  enable  the  Divisionalized  Form  to  keep  rework  down  in  predictable 
environments  (e.g.,  hierarchy) — that  is,  drawing  the  best  from  each 
organizational  form — then  we  would  establish  the  capability  to  design  an 
organization  that  is  tailored  specifically  to  a  particular  environment. 
Further,  if  we  can  identify  the  model  parameters  that  make  each  of  the 
various  organizational  forms  more  or  less  effective  in  terms  of  responses  to 
manipulations  such  as  enhanced  network  architecture  and  increased 
professional  competency,  then  we  would  establish  the  capability  to  design  an 
organization  that  is  tailored  specifically  to  a  particular  manipulation.  This 
represents  the  objective  of  articulating  the  organization  design  space:  to 
facilitate  organizational  design  specific  to  particular  environments  and 
managerial  manipulation. ” 

We  have  been  developing  new,  state-of-the  art  experimental  designs  that  allow 
analysts  to  achieve  these  goals  with  guaranteed  correctness.  For  example,  Controlled 
Sequential  Bifurcation  (CSB)  is  a  new  group  screening  method  proposed  by  Wan, 
Ankenman  and  Nelson  (2007).  The  user  specifies  two  thresholds.  Factor  effects  with 
true  magnitudes  below  the  lower  threshold  are  considered  unimportant,  those  with 
magnitudes  above  the  upper  threshold  are  considered  critically  important,  and  those  with 
magnitudes  between  the  two  thresholds  are  in  between.  Of  course,  since  the  true  effect 
magnitudes  are  unknown,  a  procedure  that  properly  identifies  the  unimportant  and 
critically  important  effects  is  needed.  CSB  does  this  by  grouping  factors  together  and 
testing  the  group’s  cumulative  effect.  If  the  group’s  effect  is  important,  then  the  group  is 
split  into  two  subgroups  for  further  testing.  Otherwise  all  factors  in  the  group  will  be 
classified  as  unimportant.  As  experiments  proceeds,  the  groups  become  smaller  and 
eventually  all  factors  that  have  not  been  classified  will  be  tested  individually.  CSB  is 
most  efficient  in  large-scale  cases  when  the  important  factors  are  sparse  and  grouped 
together  since  unimportant  factors  will  be  eliminated  in  groups  in  early  stage.  On  the 
other  hand,  in  CSB  the  signs  of  the  effects  have  to  be  assumed  pre-known,  otherwise  the 
effects  may  cancel  each  other  in  a  group  and  the  classifying  results  can  be  misleading. 

A  related  technique  is  called  FFCSB,  for  Fractional  Factorial  Controlled  Sequential 
Bifurcation  (Sanchez,  Wan  and  Lucas  2005,  2008;  see  also  Oh  2007).  FFCSB  begins 
with  a  resolution  III  fractional  factorial  design  to  separate  factors  into  positive  and 
negative  groups.  Within  each  group,  the  factors  are  ranked  based  on  their  estimated 
effects  and  CSB  is  conducted  on  the  ranked  factors  for  classification.  Numerical 
evaluation  shows  that  FFCSB  is  not  only  more  effective  than  CSB  when  the  factors’ 


effects  are  of  different  signs,  but  also  more  efficient  in  many  cases  even  when  the  CSB 
assumption  of  all  factor  effect  directions  known  a  priori  is  satisfied  since  the  benefit  of 
ranking  within  each  group  outweigh  the  additional  computational  effort  required  by  the 
resolution  III  design.  The  structure  of  the  FFCSB  procedure  is  provided  in  Figure  1. 
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Figure  1.  Structure  of  FFCSB  (from  Sanchez,  Wan  and  Lucas  2008) 

The  authors  of  this  paper  are  all  affiliated  with  the  SEED  Center  for  Data  Farming 
at  the  Naval  Postgraduate  School.  The  mission  of  the  SEED  Center  is:  “Advance  the 
collaborative  development  and  use  of  simulation  experiments  and  efficient  designs  to 
provide  decision  makers  with  timely  insights  on  complex  systems  and  operations.” 
SEED  stands  for  simulation  experiments  and  efficient  designs,  while  data  farming  refers 
to  the  process  of  “growing”  data  via  simulation.  Links  to  papers,  student  theses,  software, 
and  other  resources  can  be  found  at  the  SEED  Center’s  web  pages  (http://harvest.nps.edu). 
The  scenarios  examined  in  more  than  50  student  theses  cover  a  range  of  application  areas, 
including  network  enabled  warfare,  future  forces,  stability  and  support  operations, 
homeland  security,  logistics,  and  more. 

In  this  paper,  we  apply  FFCSB  to  identify  important  factors  in  a  Hierarchy  model 
that  drive  the  measure  of  performance  of  Project  Duration.  This  Hierarchy  model  is 
representative  of  the  prevalent  structure  in  militaries  and  serves  as  a  benchmark  for 
comparisons  of  new  organizational  forms.  The  model  is  studied  with  factor  ranges 
spanning  two  contrasting  mission-environmental  contexts:  the  Industrial  Age  and  the  21st 
Century.  This  application  appears  in  Chapter  4  of  Oh’s  (2007)  master’s  thesis. 


2.  EXPERIMENTATION  TOOLS  FOR  ORGANIZATION  THEORY 


2.1.  POW-ER  Computational  Experimentation  Tool 

These  complex  computer  simulations  of  organizational  behavior  are  developed  in  POW¬ 
ER — Projects,  Organizations  and  Work  for  Edge  Research — a  virtual  environment  for 
computational  modeling  of  C2  organizations  and  processes.  POW-ER  builds  upon 
collaborative  research  and  development  between  the  Center  for  Edge  Power  at  NPS  and 
faculty  at  Stanford  University.  The  organizational  models  are  fonnulated  from  well- 
accepted  organizational  theory.  The  computation  tool  has  been  validated  extensively  and 
thoroughly  via:  “1)  internal  validation  against  micro-social  science  research  findings  and 
against  observed  micro-behaviors  in  real-world  organizations,  2)  external  validation 
against  the  predictions  of  macro-theory  and  against  the  observed  macro-experience  of 
real-world  organizations,  and  3)  model  cross-docking  experiments  against  the  predictions 
of  other  computational  models  with  the  same  input  data  sets”  (Orr  and  Nissen  2006,  p.  8; 
Levitt  et  ah,  2005).  The  POW-ER  environment  uses  agent-based  simulation  to  emulate 
micro-behaviors  (e.g.,  trust,  learning,  skill  sets  compatibility,  skill  competency, 
centralization)  and  discrete-event-simulation  to  emulate  processes  (e.g.,  meetings, 
exception  occurrences,  rework,  process  quality).  Organizational  performance  is 
measured  by  quantitative  metrics  such  as  project  duration,  project  risk,  and  project  cost. 

2.2.  Computational  Experimentation  for  Organizational  Studies 

Using  the  POW-ER  environment,  several  researchers  have  conducted  modeling, 
simulation  and  analysis  of  comparative  perfonnance  of  alternate  C2  approaches, 
including  different  organization  structures,  work  processes,  technologies  and  personnel. 
Research  and  experimentation  results  have  been  published  in  a  series  of  recent  works. 
First,  Nissen  (2005)  laid  the  fundamentals  by  defining  the  Hierarchy  and  Edge 
organization  models  from  theory  and  comparing  their  performance  in  the  Industrial  Age 
and  21st  Century  mission  contexts.  Second,  Orr  and  Nissen  (2006)  defined  four  more 
organization  models  and  compared  the  perfonnance  of  the  six  organizations  in  the 
Industrial  Age  and  21st  Century  mission  contexts.  Third,  Gateau  et  ah,  (2007)  articulated 
an  organizational  design  space,  using  only  three  parameters  of  centralization,  hierarchy 
and  application  experience  to  characterize  organization  models.  Most  recently, 
Mackinnon  et  ah,  (2007)  calibrated  and  compared  the  impact  of  learning  and  forgetting 
micro-behaviors  on  the  Hierarchy  and  Edge  organizational  models  in  the  Industrial  Age 
and  2 1st  Century  mission  contexts 

2.3.  FFCSB:  An  Alternative  Approach  to  Tackle  the  Same  Question 

The  Hierarchy  organization  model  is  modeled  by  three  sets  of  structural  factors:  (1) 
organization  structure  (2)  communication  structure  (3)  work  structure  (Nissen  2005,  p. 
11).  The  Industrial  Age  and  21st  Century  mission  contexts  are  modeled  by  three 
manipulations  of  mission  factors:  (1)  mission  and  environmental  context,  (2)  network 
architecture  and  (3)  professional  competency  (Nissen  2005,  p.  14). 


Researchers  typically  used  full  factorial  experimental  designs  to  explore 
organizational  perfonnance  over  different  organizational  structures  and  mission  contexts. 
Nissen  (2005)  used  a  2  organizations  x  2  scenarios  design,  while  Orr  and  Nissen  (2006) 
used  a  larger  6  organizations  x  2  scenarios  x  4  manipulations  design.  Mackinnon  et  al., 
(2007)  keeps  simulation  parameters  constant  between  the  Hierarchy  and  other 
organizations  in  order  to  isolate  performance  change  due  to  learning  and  forgetting 
micro-behaviors  only.  Given  that  there  are  hundreds  or  thousands  of  factors  in  such 
complex  organization  models,  it  is  computationally  expensive  or  infeasible  to  conduct 
full  factorial  designs  on  individual  factors.  Instead,  the  design  of  experiments  would  use 
the  six  groups  of  factors  listed  above  and  change  multiple  factors  within  a  group  as  one 
variation.  Experimental  results  of  organizational  performance  were  analyzed  over  the 
entire  organization’s  model  changes  and  mission  changes  (Nissen  2005)  or  single  block 
change  (Orr  and  Nissen  2006,  Mackinnon  et  al.,  2007).  Through  analyzing  the  relative 
impact  of  each  variation  on  individual  organization  performance,  the  researchers  drew 
practical  insights.  For  instance,  Orr  and  Nissen  inferred  that:  “professional  competency 
improvements  to  the  Hierarchy/Machine  Bureaucracy  can  produce  even  more  dramatic 
results  in  terms  of  agility  as  those  associated  with  adopting  the  Edge  organizational  form. 
Hence,  a  change  in  professional  competency  can  be  substituted  to  a  large  degree  for  a 
change  in  organizational  form.  Unlike  the  substitution  effects  noted  above  for  the 
network  architecture  manipulation,  however,  the  converse  does  not  hold  for  professional 
competency:  changing  organizational  form  does  not  compensate  for  a  reversion  to  an 
efficiency-oriented  organization  and  knowledge-flow  approach”  (2006,  p.  16). 

FFCSB  offers  an  alternative  approach  to  tackle  the  same  question.  It  offers  single 
factor  resolution  and  allows  researchers  to  probe  questions  such  as:  What  are  the  most 
important  factors,  either  organizational  or  mission,  driving  the  measure  of  performance  in 
an  organizational  model?  Without  group  screening  algorithms,  it  would  have  required  an 
exorbitant  amount  of  experimentation  resources  to  conduct  full  factorial  experiments  to 
identify  performance  enhancement  (or  deterioration)  due  to  single  factors.  FFCSB 
overcomes  this  limit  by  efficient  division  and  experimentation  of  the  entire  factor  space, 
and  gradually  limiting  the  scope  of  search  for  important  factors.  Through  group 
screening  of  singular  factors,  FFCSB  can  shed  light  on  significant  individual  factors 
within  each  structural  or  mission  factor  block  that  have  the  most  impact  on  the  outcome 
of  interest. 

3.  MODEL  DESCRIPTION  &  SIGNIFICANCE 
3.1.  Hierarchy  Organizational  Model 

Figure  2  is  a  screen-capture  of  the  Hierarchy  model  in  the  POW-ER  environment.  The 
figure  illustrates  the  personnel  hierarchy  and  mission  structure  in  the  Hierarchy  model. 
Personnel  are  grouped  and  communicate  over  a  3-tier  command  chain,  which  emulates 
the  Command,  Coordination  and  Operations  levels  in  a  Joint  Task  Force  Hierarchy 
(Nissen  2005).  There  are  four  tasks  executed  sequentially  via  two  phases.  Tasks  are 
linked  to  each  other  and  to  project  milestones.  Tasks  can  flow  completed  work  down  the 
chain,  or  flow  rework  (additional  work  to  rectify  earlier  mistakes).  Personnel  are  linked 
to  work  on  meetings  and  tasks.  Operations  level  personnel  act  directly  on  tasks,  while 


Command  and  Coordination  level  personnel  act  directly  upon  their  specialized  tasks 
while  indirectly  supporting  operations  tasks. 


Figure  2.  Hierarchy  Organizational  Model  in  POW-ER  (from  Oh,  2007) 

3.2.  Measure  of  Performance:  Project  Duration 

Earlier  quoted  works  compared  organizational  performances  using  seven  measures  of 
performance  (MOPs):  duration,  cost,  project  risk,  maximum  backlog,  work  volume, 
rework  volume  and  coordination  volume.  This  FFCSB  application  focuses  on  the  first 
MOP  of  interest:  (Project)  Duration.  Duration  is  defined  as  “the  predicted  time  to 
perform  a  project,  in  working  days,  which  includes  both  direct  and  indirect  (i.e., 
coordination,  rework  and  decision  latency)  work”  (Orr  and  Nissen,  2006). 

3.3.  Factor  Exploration  Space 

Table  1  lists  the  factors  identified  in  the  Hierarchy  model  for  the  FFCSB  application, 
conducted  at  the  International  Data  Farming  Workshop  15  in  Singapore  in  November 
2007.  A  team  of  four  international  data-farming  enthusiasts  collaborated  on  the 
simulation  and  analysis  of  this  exploration  for  a  week.  In  order  to  keep  within  the 


computation  resources  and  time  constraints  for  this  section,  the  entire  factor  space  was 
divided  into  three  subspaces  for  separate  FFCSB  exploration.  Hence,  three  smaller  and 
faster  explorations  were  conducted  instead  of  one  big  exploration.  The  division  of  the 
factor  space  followed  the  three  manipulations  of  mission  context  factors:  (1)  mission  and 
environmental  context,  (2)  network  architecture  and  (3)  professional  competency.  In 
addition,  the  three  sets  of  structural  factors:  (1)  organization  structure  (2)  communication 
structure  and  (3)  work  structure  were  subsumed  under  these  factor  subspaces.  This 
division  of  factor  space  was  intended  to  mirror  that  in  the  literature  as  closely  as  possible, 
but  was  not  exact.  The  factor  ranges  of  exploration  were  derived  from  the  default  values 
of  the  Hierarchy  model  in  the  contrasting  mission  contexts  of  Industrial  Age  and  21st 
Century  (see  Appendix  for  details).  In  lieu  of  requesting  SMEs  to  specify  thresholds,  we 
selected  these  based  on  the  range  of  effects  observed  in  some  preliminary  experiments. 


Mission  &  Environment 

Network  Architecture 

Professional 

Competency 

(Project)  Function 
Exception  Probability 

(Project)  Project 

Exception  Probability 

(Task)  Effort 

(Task)  Learning  Days 

(Task)  Priority 

(Task)  Requirement 
Complexity 

(Task)  Solution 
Complexity 

(Task)  Uncertainty 

(Personnel)  Full  Time 
Equivalent 

(Personnel-Task) 

Allocation 

(Task-Task)  Successor 

(Project)  Priority 

(Project)  Length  Of  Work-day 

(Project)  Length  Of  Work-week 

(Project)  Centralization 

(Project)  Matrix-strength 

(Project)  Communication 
Probability 

(Project)  Noise  Probability 

(Project)  Instance  Exception 
Probability 

(Meeting)  Priority 

(Meeting)  Duration 

(Personnel-Meeting)  Allocation 

(Task-Task)  Rework  Strength 

(Project)  Team 

Experience 

(Personnel)  Culture 

(Personnel)  Role 

(Personnel)  Application 
Experience 

(Personnel)  Cultural 
Experience 

(Personnel)  Skill 

Ratings 

Table  1.  Factor  Space  for  Exploration  of  Hierarchy  Model 

FFCSB  was  also  applied  to  the  Hierarchy  model  with  this  entire  factor  space  in  one 
exploration.  However,  this  single  exploration  took  weeks  to  run,  without  yielding  results. 
The  sequential  nature  of  FFCSB  meant  that  the  experiments  could  not  be  parallelized. 
There  were  unusually  long  simulation  times  of  the  Hierarchy  model,  possibly  due  to 
combinations  of  factors  that  were  either  unreasonable  or  stressed  the  model  too  much. 


3.4.  Expert  Opinion  on  Significant  Factors 


Among  the  factors  identified  for  exploration,  subject  matter  experts  (SMEs)  identified  the 
following  as  important  before  the  experiments  began. 

1 .  Mission  &  Environment 

a.  (Personnel)  Full  Time  Equivalent 

b.  (Task)  Effort 

2.  Professional  Competency 

a.  (Personnel)  Application  Experience 

b.  (Personnel)  Skill  Ratings 

3.5.  FFCSB  Findings  On  Significant  Factors 

Tables  2  and  3  summarize  the  FFCSB  findings  of  important  factors  in  the  Hierarchy 
model  that  impact  Project  Duration  most.  There  were  no  factors  classified  as  important 
in  the  Network  Architecture  factor  subspace. 


Object 

Attribute 

Factor  Effect  on  Duration 

Mission 

Project  Exception  Probability 

+ 

Surface  Missions 

Effort 

+ 

Surface  Missions 

Solution  Complexity 

+ 

Ground  Missions 

Effort 

+ 

Ground  Missions 

Requirement  Complexity 

+ 

Ground  Missions 

Solution  Complexity 

+ 

Table  2.  Important  Factors  in  Mission  &  Environment  Factor  Subspace 


Object 

Attribute 

[Factor  Effect  on  Duration 

Mission 

Team  Experience 

+ 

Air  A  (Personnel) 

Skill  Ratings 

_ 

Ground  (Personnel) 

Skill  Ratings 

- 

Table  3.  Important  Factors  in  Professional  Competency  Factor  Subspace 


4.  DISCUSSION 


Before  we  discuss  the  results,  a  few  general  comments  are  in  order.  First,  all  factors 
effects  correspond  to  the  impact  on  the  MOP  of  changing  that  factor  from  its  lowest  to  its 
highest  value.  Widening  the  range  for  a  factor  deemed  unimportant  in  our  experiment 
might  make  it  show  up  as  important,  while  narrowing  the  range  for  a  factor  deemed 
important  in  our  experiment  might  make  it  drop  out.  Similarly,  an  analyst  using  more 
stringent  thresholds  than  ours  to  define  what  constitutes  an  important  factor  would  tend  to 
see  fewer  factors  identified  as  important,  while  an  analyst  using  a  less  stringent  threshold 
would  tend  to  see  more.  The  goal  of  this  study  is  not  to  provide  a  definitive  assessment 
of  how  the  Hierarchy  model  behaves  in  general,  but  rather  to  show  that  a  large-scale 
screening  experiment  can  provide  a  rich  set  of  insights  into  the  model’s  behavior.  As 
well  as  a  means  of  confirming  or  refuting  specific  hypotheses  developed  a  priori,  this  can 
be  used  to  focus  discussion,  generate  additional  hypotheses,  or  explore  how  robust  the 
organizational  performance  is  to  variations  in,  say,  the  environment  or  the  task. 

4.1.  Comparison  of  SME  Opinion  and  FFCSB  Results 

In  the  first  factor  subspace  of  Mission  &  Environment,  SMEs  identified  the 
factors  of  Full  Time  Equivalent  (FTE)  and  Effort  as  important.  FTE  measures  the 
equivalent  of  manpower  resources  available  and  Task  Effort  quantifies  the  time  effort 
requirement  of  the  task.  Contrary  to  expert  opinion,  FFCSB  does  not  classify  any  FTE 
factors  as  important  over  the  factor  range  of  exploration.  Thus,  FTE  is  not  as  important 
as  the  other  factors  in  this  subspace  in  impacting  the  Project  Duration.  This  is  an 
interesting  finding,  particular  since  FTEs  were  varied  over  a  wide  range  (from  one -half  to 
twice  the  default  number  for  each  personnel  category).  The  implications  here  are  that  the 
organizational  perfonnance  is  robust  to  the  loss  of  FTE  in  any  single  personnel  category, 
and  that  adding  more  manpower  to  any  single  group  also  has  relatively  little  impact  on 
project  duration.  Further  experimentation  could  be  used  to  confirm  these  results,  or  to 
estimate  the  net  impact  of  simultaneous  FTE  changes  in  two  or  more  teams. 

In  line  with  expert  opinion,  FFCSB  classifies  Effort  factors  as  important,  but  of 
the  eight  possible  missions,  only  those  for  Surface  Missions  and  Ground  Missions  are 
flagged.  Critical  path  analysis  of  the  Hierarchy  model  explains  why  factors  associated 
with  only  these  two  missions  showed  up  consistently  as  important.  The  red  bars  in  Figure 
3  depict  the  critical  path  of  the  project  simulated  in  the  Hierarchy  model.  Following  the 
red  bars,  the  Air  Missions  1,  Surface  Missions  and  Ground  Missions  are  on  the  critical 
path.  Of  these  three  missions,  the  Surface  Missions  and  Ground  Missions  have  minimum 
float,  i.e.,  there  is  no  allowance  for  shifting  these  missions  in  time.  Hence,  these  two 
missions  are  crucial  to  the  MOP  of  Project  Duration.  Besides  the  Task  Effort  factor, 
FFCSB  also  classified  the  Solution  Complexity  factors  of  the  Surface  and  Ground 
Missions  as  important,  as  well  as  the  Requirements  Complexity  of  the  Ground  Missions. 
Thus,  FFCSB  has  further  quantified  expert  opinion  by  flagging  only  factors  associated 
with  missions  on  the  critical  path  with  specific  characteristics. 


Gantt  Chart  -  Project  Mission 
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Figure  3.  Critical  Path  Analysis  of  Hierarchy  model  shows  Air  Missions  1,  Surface 
Missions  and  Ground  Missions  on  Critical  Path  (Best  viewed  in  color) 

In  addition,  FFCSB  classifies  the  global  factor  of  Project  Exception  Probability 
(PEP)  as  important.  PEP  is  the  probability  that  a  subtask  will  fail  and  generate  rework 
for  failure  dependent  tasks.  This  factor  is  significant  for  the  Hierarchy  model  that  is 
characterized  by  sequential  and  interdependent  tasks  and  hence,  suffers  a  longer  Project 
Duration  in  the  event  of  increased  PEP. 

In  the  second  factor  subspace  of  Network  Architecture,  there  are  no  factors 
classified  as  important.  This  finding  is  in  agreement  with  SMEs,  who  did  not  expect  any 
important  factors  in  this  subspace.  A  set  of  (relatively  computationally  expensive) 
resolution  V  fractional  factorial  design  (allowing  the  estimation  of  both  main  effects  and 
two-way  interactions)  was  used  to  verify  the  factor  coefficients  in  this  factor  group. 
(Sanchez  and  Sanchez  (2005)  provide  a  simple  method  for  generating  resolution  V 
fractional  factorials  for  very  large  numbers  of  factors;  code  in  the  fonn  of  an  executable 
jar  file  is  available  at  the  SEED  Center  for  Data  Farming  web  pages.)  The  results 
confirmed  that  the  factor  coefficients  were  relatively  small  in  magnitude  and  hence, 
practically  insignificant.  We  remark  that  Network  Architecture  is  the  area  where  the 
majority  of  the  money  and  effort  have  gone  in  pursuit  of  developing  net-centric  or  net- 
enabled  forces.  It  is  interesting  that  both  our  SMEs  and  our  FFCSB  results  agree  that 
Network  Architecture  is  unimportant.  These  results  suggest  that,  at  least  for  Hierarchical 
C2  structures  such  as  the  one  we  study,  a  shift  in  focus  to  examine  the  mission 
environments  and  individual  skill  levels  would  be  beneficial. 

In  the  third  factor  subspace  of  Professional  Competency,  experts  identified  Skill 
Ratings  and  Application  Experience  factors  as  important.  FFCSB  classified  the  Skill 
Ratings  of  the  Air  A  and  Ground  personnel  as  important,  but  not  that  of  the  Surface 


personnel.  These  three  groups  of  personnel  are  operations  personnel  and  directly 
responsible  for  the  missions  on  the  critical  path.  The  contrast  between  the  three  missions 
is  that  the  Surface  Missions  require  a  considerably  longer  effort  of  2 1  months  versus  that 
of  the  Ground  Missions  (6.5  months)  and  Air  Missions  1  (11  months).  These  findings 
suggest  that  Skill  Levels  may  be  more  critical  for  missions  that  lie  on  the  critical  path  and 
have  relatively  shorter  Effort  requirements.  FFCSB  did  not  classify  Application 
Experience  as  important. 

However,  interestingly,  FFCSB  classified  Team  Experience  as  important  and 
positively  related  to  the  MOP.  Team  Experience  quantifies  the  degree  of  familiarity  that 
team  members  have  in  working  with  one  another  as  a  team.  In  other  words,  this  finding 
suggests  that  more  team  experience  leads  to  longer  Project  Duration  in  the  Hierarchy 
model.  This  result  seems  counter-intuitive.  In  fact,  in  two  workshops  subsequent 
workshops  where  participants  were  shown  the  factor  categories  in  Table  1  and  asked  to 
identify  factors  they  felt  were  important,  Team  Experience  was  chosen  as  a  way  to 
decrease  project  duration  time!  Yet  this  seemingly  counter-intuitive  finding  may  have 
been  observed  in  earlier  research  and  experimentation.  Ramsey  and  Levitt  (2005) 
summarized  high  level  findings  from  Horii,  Jin  and  Levitt’s  “Modeling  and  Analyzing 
Cultural  Influences  on  Team  Performance  through  Virtual  Experiments”  (2004)  on  the 
impact  of  cultural  differences  in  project  teams:  “Japanese-style  organizations  were  more 
effective,  with  either  US  or  Japanese  agents,  at  performing  tasks  with  high 
interdependence  when  the  team  experience  of  members  was  low.”  The  Hierarchy  model 
studied  in  this  application  shares  common  characteristics  of  centralized  authority,  high 
formalization,  and  multiple  hierarchies  with  the  Japanese-style  organization  modeled  in 
Horii,  Jin  and  Levitt  (2004,  pp.  3).  In  addition,  these  experiments  had  used  the  MOPs  of 
Project  Duration  and  Quality  Risk  to  quantify  team  performance,  while  this  FFCSB 
application  only  used  Project  Duration.  Hence,  there  is  common  ground  to  compare  the 
similarity  of  both  findings.  Had  the  original  intuition  on  Team  Experience  been  applied 
with  conventional  screening  algorithms,  this  factor  could  have  distorted  screening 
findings. 

Lastly,  there  were  two  general  observations  of  interest.  First,  there  were  more 
important  factors  associated  with  the  Operations  layer  of  the  JTF  structure  than  the  other 
layers.  Recall  that  the  Hierarchy  model  has  a  3-tier  command  chain  that  models  the 
Command,  Coordination  and  Operations  layers  in  a  JTF.  Second,  there  were  more 
uncontrollable  or  difficult  to  control  factors  (e.g.,  Project  Exception  Probability,  Task 
Requirement  Complexity,  Task  Solution  Complexity  and  Team  Experience)  than 
controllable  or  easy  to  control  factors  (e.g.,  Skill  Ratings.) 

4.2.  Choice  of  Screening  Method 

The  important  factor  classification  and  observations  are  meant  to  provide 
direction  for  researchers  in  future  work  and  optimize  their  experimentation  budget  on 
truly  important  factors.  This  first-case  FFCSB  application  on  a  real-world  simulation 
model  has  produced  results  that  are  coherent  with  critical  path  analysis  and  that  agree 
with  earlier  research  on  similar  models.  Hence,  it  is  an  encouraging  sign  that  FFCSB  can 
serve  as  a  complementary  tool  to  better  understand  complex  simulation  models.  Of 


course,  these  findings  are  preliminary  and  apply  to  a  specific  hierarchical  C2  structure: 
care  should  be  taken  in  drawing  general  conclusions. 

FFCSB  is  not  the  only  potential  experimental  design  that  can  be  applied  to 
complex  simulation  models.  Other  experimental  designs  are  also  suitable  for  these  types 
of  applications,  and  further  methodological  work  is  currently  underway.  A  variant  called 
FFCSBX  is  useful  for  categorizing  main  effects  even  in  the  presence  of  two-way 
interactions  (Sanchez  et  ah,  2008).  Another  screening  approach  uses  sequential  fractional 
factorial  designs  that  are  typically  more  efficient  than  a  single-stage  fractional  factorial 
design  (Shen  and  Wan  2005).  A  hybrid  approach  allows  the  analyst  to  estimate  factor 
effects  (rather  than  simply  classify  factors  as  important  or  unimportant)  at  the  completion 
of  the  experiment  (Wan  et  ah,  2008).  In  addition,  the  new  DOE-based  algorithm  of 
Chang  et  al.  (2007),  called  Stochastic  Trust  Region  Gradient-Free  Method  (STRONG), 
for  solving  large-scale  simulation  optimization  problems,  appears  particularly  promising 
because  it  is  easy  to  automate  and  yet  has  provably  reliable  asymptotic  performance. 
Regardless  of  the  screening  procedure  used,  the  analyst  may  wish  to  follow  up  with 
further  experiments  that  examine  those  factors  deemed  important  in  more  detail. 

5.  CONCLUDING  REMARKS 

In  this  paper,  we  illustrate  how  an  efficient  experimental  design  approach  can  support 
current  research  in  Computation  Organization  Theory.  The  FFCSB  application  produced 
many  delightful  surprises.  Part  of  the  important  factor  classification  was  in  line  with 
expert  opinion  and  part  of  it  ran  contrary  to  expectations.  There  were  new  findings  of 
important  factors  that  were  justified  by  critical  path  analysis  and  in  agreement  with 
earlier  research  and  experimentation.  Overall,  this  particular  FFCSB  application  has 
confirmed  expert  opinion,  flagged  out  new  important  factors  and  produced  some 
interesting  hypothesis,  all  for  further  exploration. 

There  are  limitations  to  the  FFCSB  application  to  any  model.  FFCSB  assumes  a 
main  effects  model  and  interactions  can  distort  the  accuracy  of  factor  classification.  The 
nature  of  the  response  variance  (homogeneous  or  heterogeneous)  and  its  magnitude  are 
unknown.  Both  model  characteristics  can  have  bearings  on  the  FFCSB  findings  and 
accuracy  guarantees.  Particular  to  the  Hierarchy  model,  the  observations  of  this  FFCSB 
exploration  are  unique  to  the  factor  space  organization  and  ranges  of  exploration.  Hence, 
the  findings  are  not  conclusive  of  the  Hierarchy  model.  The  important  factor 
classification  and  observations  are  meant  to  provide  direction  for  researchers  in  future 
work  and  optimize  their  experimentation  budget  on  truly  important  factors.  This  first- 
case  FFCSB  application  on  a  real-world  simulation  model  has  produced  results  that  are 
coherent  with  critical  path  analysis  and  that  agree  with  earlier  research  on  similar  models. 
Hence,  it  is  an  encouraging  sign  that  FFCSB  can  serve  as  a  complementary  tool  to  better 
understand  complex  simulation  models. 

Continued  exploration  of  the  Hierarchy  model  with  different  factor  space 
organization  and  factor  ranges  would  form  a  good  sensitivity  analysis  study  of  the 
FFCSB  application  on  the  model.  Exploring  an  Edge  organization  model  would  fonn  an 


interesting  study  in  itself,  and  allow  for  meaningful  contrasts  between  the  competing 
organizational  forms. 

The  benefits  of  being  able  to  easily  perform  a  screening  experiment  on  a  complex 
organizational  model  cannot  be  overstated.  In  the  absence  of  this  capability,  an  analyst 
must  either  limit  themselves  to  a  small  number  of  factors  to  investigate,  or  make  changes 
to  a  large  number  of  factors  simultaneously  to  come  up  with  a  small  number  of 
organizational  forms,  settings,  or  task  types  to  investigate.  We  remark  that  the  POW-ER 
model  and  its  predecessor,  VDT  (Virtual  Design  Team  Research  Group,  2006)  have  been 
successfully  used  in  practice  for  over  a  decade.  Although  this  model  that  has  been 
“validated”  by  a  history  of  successful  applications  in  the  field,  it  is  nonetheless  difficult 
for  experts  to  fully  grasp  the  complex  interplay  of  the  complete  set  of  potential  factors. 
This  is  particularly  important  in  command  and  control  research,  as  we  seek — not  to 
model  existing  organizations  and  organizational  structures — but  to  define  new  ones  that 
will  be  effective  for  our  military  transformation. 

Screening  experiments  also  offer  opportunities  to  validate  a  model  for  a  particular 
use.  For  example,  if  results  contradict  SME  opinion  and,  after  further  discussion  or  field 
experiments,  the  model  results  are  shown  to  be  inaccurate,  the  model  should  be  modified. 
Controversial  results  from  a  screening  experiment  may,  in  fact,  help  identify  alternatives 
that  merit  testing  in  the  field.  In  the  long  run,  this  cycle  of  model-test-model  will  lead  to 
models  that  provide  better  representations  of  reality,  as  well  as  a  better  understanding  of 
the  model’s  behavior,  strengths,  and  limitations. 

In  summary,  there  are  efficient  experimental  designs  and  screening  approaches  that 
are  easy  to  implement,  require  fewer  assumptions  than  conventional  experimental  design 
methods,  and  yet  can  provide  analysts  with  better  insights  when  the  experiments  are 
complete.  These  can  substantially  reduce  the  computational  requirement  for  military 
leadership  to  identify  optimal  factor  combinations,  and  lead  to  a  much  broader  and  deeper 
understanding  of  the  system.  This  will  facilitate  the  decision  making  process 
dramatically. 
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APPENDIX 


Table  4  lists  the  factors  identified  in  the  Hierarchy  model  for  FFCSB  application  and 
their  range  of  exploration. 


Object 

[Factor 

Organizational 

Structure 

FFCSB  Exploration 

Industrial 

Age 

Hierarchy 

21st 

Century 

Hierarchy 

Low 

value 

High 

value 

Project 

Priority 

Medium 

Medium 

Low 

High 

Work-day 

480 

480 

360 

600 

Work- week 

2400 

2400 

1440 

3600 

Team  Experience 

Low 

Low 

Low 

Medium 

Centralization 

High 

High 

Medium 

High 

Formalization 

High 

High 

Medium 

High 

Matrix  Strength 

Low 

Low 

Low 

Medium 

Communication 

Probability 

0.1 

0.1 

0.05 

0.2 

Noise  Probability 

0.3 

0.3 

0.01 

0.6 

Functional 

Exception 

Probability 

0.1 

0.2 

0.05 

0.4 

Project  Exception 
Probability 

0.1 

0.2 

0.05 

0.4 

Instance 

Exception 

Probability 

0 

0 

0.01 

0.4 

Meeting 

Priority 

High 

High 

Medium 

High 

Duration 

2  hours 

2  hours 

0.5  hours 

4  hours 

Personnel 

Culture 

Generic 

Generic 

American 

Japanese 

Role 

(Various) 

(Same) 

PM 

ST 

Application 

Experience 

Medium 

Low 

Low 

Medium 

Cultural 

Experience 

Medium 

Medium 

Low 

High 

Full  Time 
Equivalent 

(Various) 

(Same) 

0.5  * 
Default 

2  * 

Default 

Skill  Ratings 

Medium 

Medium 

Low 

High 

Table  4.  Factors  &  Ranges  in  Hierarchy  Model  for  FFCSB  Application 


Object 

Factor 

Organizational 

Structure 

FFCSB  Exploration 

Industrial 

Age 

Hierarchy 

21st 

Century 

Hierarchy 

Low 

value 

High 

value 

Task 

Effort 

(Various) 

(Same) 

0.5  * 
Default 

2  * 

Default 

Learning  Days 

0 

0 

0 

90 

Priority 

Medium 

Medium 

Low 

High 

Requirement 

Complexity 

Medium 

High 

Medium 

High 

Solution 

Complexity 

Medium 

High 

Medium 

High 

Uncertainty 

Medium 

High 

Medium 

High 

Meeting 

Assignment 

0.1-1.0 

0.1 

1.0 

Task 

Assignment 

Allocation 

0. 9-1.0 

1.0 

0.7 

1.0 

Successor 

Time  Lag 

0 

0 

0.0  pet- 
complete 

IHlfg 

Rework 

Strength 

(Various) 

0.15,0.3,1.0 

0.1 

0.15 

0.3 

Table  4  (contd). 
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